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LONG-TERM  GOALS 

A  fully  coupled  global  atmosphere/ocean/ice/wave/land  prediction  system  providing  daily  predictions 
out  to  10  days  and  weekly  predictions  out  to  30  days. 

OBJECTIVES 

The  first  objective  is  designing  the  infrastructure  for  operational  implementation  of  the  fully  coupled 
global  system.  This  design  includes  scoping  the  system  to  fit  within  the  constraints  of  the  operational 
centers.  These  constraints  include  both  computational  and  manpower  resources.  Determination  of  the 
run  cycle  will  take  into  account  the  three  main  forecasts  (10-day  deterministic,  30-day  detenninistic 
and  90-day  probabilistic)  with  data  assimilation  for  the  different  components  to  be  considered.  Data 
streams,  inputs  and  post-processing  also  will  be  considered. 

The  second  objective  is  the  validation  of  the  three  primary  forecasts.  The  validation  will  be  conducted 
relative  to  traditional  metrics  that  have  been  used  in  prior  transitions  of  individual  ESPC  components. 
In  addition,  the  ESPC  forecasts  will  be  validated  against  the  ESPC  demonstration  goals. 

APPROACH 

Work  area:  Design  infrastructure  for  operational  implementation  for  coupled  system 

The  supporting  infrastructure  development  will  begin  to  cycle  with  the  data  assimilation  of  each 
component,  provide  the  appropriate  daily  hindcasts,  include  the  analysis  insertion  as  appropriate.  The 
first  step  will  be  a  loosely  coupled  DA  system  with  data  assimilation  carried  out  separately  for 
individual  components  at  the  initial  time.  The  exchange  variables  between  components  will  be 
adjusted  to  ensure  consistency  and  balance  as  necessary.  This  will  be  controlled  by  the  scheduling  of 
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individual  jobs.  As  the  truly  coupled  DA  system  developed  under  separate  projects  become  available 
and  maturing,  it  will  be  integrated  into  the  full  execution  of  the  total  system,  ready  for  the  ESPC 
demonstration. 

Work  area:  Validation  testing  relative  to  prior  metrics  and  ESPC  Demonstration  goals 

Define  a  scorecard  of  validation  metrics  for  all  components  of  the  coupled  system.  Initially  perform 
validation  testing  of  the  coupled  system  using  existing  metrics,  however,  definition  of  new  metrics 
may  be  required  to  examine  the  exchange  processes  at  the  various  interfaces.  Conduct  validation 
relative  to  the  national  ESPC  demonstrations. 


WORK  COMPLETED 


Work  area:  Design  infrastructure  for  operational  implementation  for  coupled  system 

Task  1:  Define  implementation  across  operational  systems,  architecture  requirements,  cycling  setup 
including  DA. 

The  operational  implementation  design  document  was  submitted  at  the  end  of  the  last  fiscal  year.  At 
the  request  of  the  sponsor,  it  was  then  sent  to  the  operational  centers  (NAVOCEANO  and  FNMOC) 
for  review  and  their  commentary  was  incorporated  into  the  document.  A  final  version  appeared  in  print 
in  February  2014  as  an  NRL  Memorandum  Report  (Metzger  et  ah,  2014). 

Task  2:  Develop  infrastructure  to  run  NAVGEM/HYCOM with  DA 


The  cycling  of  the  coupled  atmosphere-ocean-ice  system  along  with  the  associated  data  assimilation  is 
defined  in  Figure  1  for  NAVGEM  and  in  Figure  2  for  HYCOM+CICE.  Currently  these  are  the  only 
model  components  that  are  working  coupled  together. 
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Figure  1:  A  schematic  of  NA  VGEM  cycling  within  the  coupled  atmosphere-ocean-ice  system.  The 
NAVDAS-AR  data  assimilation  will  occur  every  six  hours.  The  initial  run  will  need  to  occur  at  -12Z 
because  that  is  the  starting  point  for  HYCOM+CICE  each  day. 
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Figure  2:  A  schematic  ofHYCOM+CICE  cycling  within  the  coupled  atmosphere-ocean-ice  system. 
The  NCODA  ocean  and  ice  analyses  will  occur  once  per  day  at  -12Z.  The  ocean  model  needs  to  run 
to  +24Z  because  NCODA  uses  the  00Z-24Z  HYCOM  daily  mean  as  the  first  guess  for  the  ocean 
analysis  and  the  24Z  instantaneous  CICE  field  as  the  first  guess  for  the  ice  analysis. 

Timing  estimates  have  been  refined  for  the  integration  of  a  NAVGEM  (T350L50)  -  NAVDAS  / 
HYCOM+CICE  (1.12°)  -  NCODA  reanalysis  using  1000  IBM  iDataPlex  processors.  The  total 
walltime  for  a  single  day  is  282  minutes,  or  71.7  days  for  a  year  of  integration. 

An  NCODA  analysis  for  WAVEWATCH  III  (WW3)  presently  runs  at  FNMOC,  but  there  is  no  wave 
analysis  running  on  the  Navy  DSRC  and  so  it  is  a  missing  data  assimilation  component  of  the  coupled 
atmosphere-ocean-ice-waves  system.  FNMOC  provided  guidance  to  NRL-SSC  on  how  to  implement 
NCODA  for  the  assimilation  of  altimeter  data  into  WW3  inputs.  Progress  has  been  made  converting 
WW3  output  files  to  an  NCODA-ready  format,  the  first  step  of  the  process.  The  question  of  altimeter 
data  availability  in  real-time  is  still  not  fully  resolved  and  is  being  investigated.  Work  on  completeing 
the  scripting  continues. 

Task  3:  Scripting  implementation 

In  the  coming  year  the  control  of  the  coupled  system  will  be  converted  from  a  specialized  script  to  a 
series  of  modular  re -runnable  tasks  for  control  by  a  scheduler  of  the  user’s  choosing.  These  tasks  will 
follow  general  good  practice  and  follow  the  strategy  used  by  other  major  operational  centers.  An 
archive  of  the  OOZ  initial  conditions  from  the  FNMOC  operational  run  will  be  maintained  on  the  Navy 
DSRC,  these  can  be  used  to  test  various  aspects  of  the  developing  system  to  ensure  their  functionality 
across  a  range  of  conditions  (states).  A  meta-schedular  suite  prototype  has  been  designed  and  is  being 
tested,  and  the  first  skeleton  has  been  validated.  The  meta-schedular  allows  for  a  greater  control  of  the 
tasks  interoperability  and  more  efficient  running  and  monitoring  of  tasks  than  using  a  “pure”  scripting 
language  such  as  korn  or  bash.  An  example  of  one  of  these  higher  level  tools  is  SMS  which  is  being 
used  opeationally  at  FNMOC.  A  diagram  of  the  ESPC  control  suite  is  shown  below  in  Figure  3,  this  is 
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produced  directly  from  the  control  file  by  the  meta-scheduler  which  in  this  case  is  called  “cylc”. 
Various  colors  indicate  the  pre-processing,  forecast,  and  post-processing  steps.  Double  lines  around 
the  fields  indicate  the  task  is  a  family  which  contains  sub-tasks  that  means  there  are  independent 
atmospheric,  ocean  and  sea  ice  components.  At  this  time  NAVGEM,  HYCOM  and  CICE  are  being 
run  but  the  addition  of  WW3  into  this  framework  is  in  prinicple  a  straightforward  incorporation. 


Figure  3:  A  graph  of  the  suite  control  file  for  the  ESPC  system.  This  diagram  was  produced  by  the 
cylc  meta-scheduler  which  is  now  being  testing  as  the  task  management  system  for  the  ESPC 
system.  The  different  colors  indicate  pre-processing  (green),  forecast  (red),  and  post-processing 
(blue),  while  white  indicates  a  non  inter-dependent  task. 

Work  area:  Validation  testing  relative  to  prior  metrics  and  ESPC  Demonstration  goals 

Task  1:  Given  the  long  term  reforecasts  constructed  in  the  ESPC  development  work,  gather  historical 
data  sets. 

This  year,  NRL-SSC  completed  a  1993-2012  Ocean  Reanalysis  using  1/12°  global  HYCOM-NCODA. 
As  part  of  the  preparatory  work  for  that  hindcast,  all  the  necessary  satellite  altimeter,  ice  concentration 
and  sea  surface  temperature  data,  in-situ  profile  data  (Argo  profiling  floats,  XBTs,  CTDs),  moored  and 
drifting  data  have  been  gathered,  quality  controlled  and  put  into  NCODA-ready  format.  Beyond  this 
period  these  data  are  routinely  gathered  to  be  used  as  part  of  the  Global  Ocean  Forecast  System 
(GOFS).  Thus,  all  the  required  input  observational  datastreams  should  be  in  place  for  the  ocean  and  ice 
components  of  a  long-tenn  coupled  reforecast. 

NRL-MRY  continues  to  archive  the  atmospheric  data  which  extends  an  archive  spanning  back  to 
November  2008.  This  atmospheric  DA  now  processes  over  100  million  observations  and  assimilates 
operationally  over  3  million.  In  the  coming  year  an  extension  of  the  archive  is  possible;  however,  the 
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time  to  gather  the  data  types  will  be  extensive  as  the  distinct  data  types  number  over  100  which 
populate  49  distinct  directories.  The  current  volume  is  roughly  32  Gb  per  day  for  these  various  data. 

A  reanalysis  covering  a  single  year  is  being  planned  which  can  be  used  to  exercise  the  various 
coupling  infrastructure  components.  A  reanalysis  employing  the  full  5  year  archive  will  be  undertaken 
as  the  various  components  and  the  forecasts  are  verified. 

Task  2:  Evaluation  of  coupled  runs  using  prior  metrics. 

We  have  begun  the  process  of  designing  integrated  skill  diagnostics  for  the  HYCOM+CICE 
component  of  ESPC,  starting  with  the  previous  ocean-ice  validation  test  reports  that  have  been  used  to 
evaluate  earlier  systems.  These  reports  are  being  used  to  compile  a  list  of  metrics  and  associated  data 
sources  for  consideration  with  this  system.  We  have  begun  researching  and  designing  oceanic  metrics 
with  the  longer  range  forecasts  (~90  days)  of  ESPC  in  mind,  instead  of  the  shorter  range  (-7-14  days) 
of  earlier  systems.  These  metrics  will  focus  on  longer  term  dynamics  such  as  key  boundary  layer 
transports,  main  thermocline  displacement  and  overturning  circulations  instead  of  the  shorter  time 
scale  upper  ocean  metrics  traditionally  used.  We  have  begun  examining  the  distribution  and  types  of 
data  routinely  available  for  assimilation  into  HYCOM  and  CICE,  in  order  to  better  to  understand  the 
challenges  in  designing  integrated  skill  diagnostics  for  the  poorly  observed  ocean.  Further  research 
using  the  existing  scientific  literature  and  discussions  with  team  members,  external  scientists 
experienced  with  coupled  modeling  and  operational  centers  are  planned  in  FY15  to  help  define  a  state 
of  the  art  approach  for  routinely  monitoring  the  perfonnance  of  the  coupled  ESPC  system. 

Task  3:  Development  of  metrics  to  ESPC  demonstrations  as  related  to  Navy  activities 

Item  IV:  Validation  of  Arctic  ice  formation/melt: 

Sea  Ice  Prediction  Network  (SIPN)  forecasts  were  completed  using  a  test  version  of  the  ESPC  global 
coupled  atmosphere-ocean-ice  system.  The  SIPN  is  a  multi-model  comparison  of  September  Arctic  sea 
ice  extent  forecasts  with  lead  times  of  30  to  90  days.  NAVGEM  versions  1.1  and  1.2  were  coupled 
with  HYCOM  version  2.2. 77i,  and  CICE  version  4.  The  forecasts  started  on  June  30th  12Z  2014  and 
July  31st  12Z  2014  with  restart  files  from  the  individual  assimilative  model  components.  No  data 
assimilation  or  cycling  was  perfonned  for  these  forecasts  (i.e.  cold  starts).  Model  horizontal  grid 
resolution  was  T359L50  for  NAVGEM  1. 1/1.2,  ~9  km  (at  the  equator)  for  HYCOM  and  ~3.5  km  (at 
the  North  Pole)  for  CICE.  Diagnostics  were  performed  against  FNMOC’s  sea  ice  concentration  data 
set,  which  is  the  same  data  set  used  for  CICE  assimilation.  The  model  forecasts  were  un-tuned,  i.e.  no 
heat  flux  corrections  were  used. 

RESULTS 

Work  area:  Validation  testing  relative  to  prior  metrics  and  ESPC  Demonstration  goals 
Task  2:  Evaluation  of  coupled  runs  using  prior  metrics. 

Through  evaluation  of  some  of  the  coupled  atmosphere-ocean-ice  experiments,  it  was  learned  that  the 
atmospheric  scorecard  will  have  to  be  modified  to  take  into  account  more  metrics  that  evaluate 
processes  at  or  near  the  air-ocean  interface.  For  example,  some  of  the  initial  coupled  30-day  forecasts 
had  a  tendency  to  develop  far  too  many  and  too  strong  tropical  cyclones  in  the  Indo-West  Pacific 
basins.  This  propensity  for  excessive  storms  was  not  detected  using  the  traditional  500  hPa  height 
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anomaly  metric  or  other  atmospheric  scorecard  metrics,  but  was  very  evident  using  a  newly  defined 
cyclonic  vorticity  at  850  hPa  metric. 

Task  3:  Development  of  metrics  to  ESPC  demonstrations  as  related  to  Navy  activities 
Item  IV:  Validation  of  Arctic  ice  formation/melt: 

The  ESPC  global  atmosphere-ocean-ice  forecasts  are  comparable  to  the  other  numerical  models  in  the 
SIPN  (Figure  4).  The  minimum  September  Arctic  sea  ice  extent  for  the  forecasts  starting  on  June  30th 
12Z  2014  was  4.3  million  km2  for  the  NAVGEM  vl.l  —  HYCOM  -  CICE  run  and  4.5  million  km2  for 
the  NAVGEM  vl.2  -  HYCOM  -  CICE  system.  Model  forecasts  with  the  start  date  of  July  31st  12Z 
2014  had  a  minimum  September  Arctic  sea  ice  extent  of  4.8  and  4.5  million  km2  for  the  system  using 
NAVGEMvl.l  and  NAVGEMvl.2  respectively.  September  sea  ice  minimum  extents  for  the  SIPN 
models  ranged  from  4. 1  to  5.6  million  km2.  According  to  the  National  Snow  and  Ice  Data  Center,  a 
2014  Arctic  sea  ice  minimum  of  5.02  million  km2  was  reached  on  19  September. 


10 


Northern  Hemisphere:  ESPC-Coupled  System 
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Figure  4:  Observed  (black)  Northern  Hemisphere/Arctic  sea  ice  extent  (Mkm2  -  defined  as  15%  ice 
concentration  and  greater)  during  the  2014  melt  season  vs.  sea  ice  extent  from  a  suite  of  coupled 
NAVGEM-HYCOM-CICE  forecasts  started  from  30  June  2014  atmosphere-ocean-ice  initial 
conditions  ( ICs )  (blue/green)  and  from  31  July  2014  ICs  (orange/burgundy).  Different  versions  of 
NAVGEM  (1.1  vs.  1.2)  were  used  in  the  forecasts.  The  black  dots  indicate  sea  ice  extent  forecasts 
from  specific  modeling  systems  and  the  gray  bar  is  the  range  for  all  the  SIPN 
modeling  systems  that  submitted  reports.  The  2014  minimum  sea  ice  extent  had  not  yet  been 

reached  when  this  graphic  was  generated. 
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Though  the  September  minimum  Arctic  sea  ice  extents  are  comparable  to  other  models,  the  initial 
decrease  of  sea  ice  at  the  beginning  of  each  forecast  may  be  an  issue  and  will  be  investigated  further. 

In  addition,  these  forecasts  were  compared  to  observations  in  the  southern  hemisphere  (SH).  SH 
analysis  suggests  that  the  coupled  system  has  a  similar  sea  ice  extent  for  the  first  five  forecast  days,  but 
unrealistic  increases  in  sea  ice  occur  on  longer  time  scales.  These  long-range  biases  may  be  caused  by 
NAV GEM’s  cold  bias  over  sea  ice  during  the  polar  night. 

IMPACT/APPLICATIONS 

ESPC  forecasts  will  provide  environmental  information  to  meet  Navy  and  DoD  operations  and 
planning  needs  throughout  the  globe  from  under  the  sea  to  the  upper  atmosphere,  and  from  the  tropics 
to  the  poles. 

RELATED  PROJECTS 

Dependence  on  outside  6.2  work:  ONR-funded  MJO  DRI;  ONR-funded  ITOP;  ONR-funded  Tropical 
Cyclone;  ONR-funded  Unified  Physical  Parameterization  DRI;  ONR-funded  Seasonal  Prediction  DRI; 
NOPP-funded  HYCOM;  NOPP-funded  waves  work;  NRL-funded  high  resolution  ice  coupling;  NRL- 
funded  MJO  6.2;  NRL-funded  Fresh  Water  Fluxes  6.2;  NRL-funded  Wave  conservation  6.2. 

Dependence  on  outside  6.4  work:  The  NUOPC  interoperability  layer  under  ESMF  is  a  critical  piece  on 
which  this  development  initially  built  upon.  Large  Scale  Atmospheric  Modeling  (under  6.4  NAV GEM) 
is  providing  extensions  and  validation  to  NAV GEM.  Large  Scale  Ocean  Modeling  is  providing 
extensions  and  validation  to  HYCOM.  Small  Scale  Ocean  Modeling  is  providing  extensions  and 
validation  to  WaveWatch  III.  Ocean  Data  Assimilation  is  providing  vertical  synthetic  covariance  and 
data  assimilation  for  HYCOM  and  new  data  streams  from  altimeter  satellites.  Atmospheric  Data 
Assimilation  is  providing  extensions  and  validation  to  NAV GEM  assimilation.  Satellite  Ice  is 
providing  new  data  streams  for  assimilation  in  CICE. 
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